Top-K Entity Resolution with Adaptive Locality-Sensitive Hashing

Given a set of records, entity resolution algorithms find allthe records referring to each entity. In this paper, we studythe problem of top-k entity resolution: finding all the recordsreferring to the k largest (in terms of records) entities. Top-kentity resolution is driven by many modern applications thatoperate over just the few most popular entities in a dataset.We propose a novel approach, based on locality-sensitivehashing (LSH), that can very rapidly and accurately pro-cess massive datasets. Our key insight is to adaptively de-cide how much processing each record requires to ascertainif it refers to a top-k entity or not: the less likely a recordis to refer to a top-k entity, the less it is processed. Theheavily reduced amount of processing for the vast majorityof records that do not refer to top-k entities, leads to sig-nificant speedups. Our experiments with web images, webarticles, and scientific publications show a 2x to 25x speedupcompared to the traditional approach for high-dimensionaldata.